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Why shrink data? 

Money! 

Reduce bandwidth costs 

Initial download / updates #profit 

Continuous connections 

Expand reach 

Decreased loading times 
Smaller app size 
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Why is it fun? 


A history of neglect 
+ 

More use cases every day 
+ 

A recent „Gold rush" 




Why is now a good time? 


CPU / Memory gap 
Fidelity / 

Bandwidth gap 

Hi res displays 
versus 

bandwidth & storage 



1980 1985 1990 1995 2000 2005 2010 

Year 

C 2007 Bsaner, Inc. At npfc moved 
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What is data compression? 


Wikipedia: 

„[...] encoding information 
using fewer bits 

than the original representation" 



III! 
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Two flavours of compression 

Lossless 

All information is retained 

Lossy 

An approximation is retained 
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History & Concepts 


Information Theory, ~1948 


Claude Shannon 
Entropy 
Shannon limit 



Pr(X= 1) 
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History & Concepts 


Prefix code, ~1952 
Variable length code 
Translated with a dictionary 
Constructed with Huffman tree 
Fast and efficient 
Still used today 
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History & Concepts 


Lempel-Ziv, 1977 

Base for the LZ-family 
Refers back to already processed data 
„Sliding Window" 

Implicit dictionary creation 




mi 
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History & Concepts 

Deflate, 1991 

LZ77 + Huffman 
Used everywhere! 
http://zlib.net 
25 years old! 






1 1 


ii 


GDC 


GAME DEVELOPERS CONFERENCE EUROPE COLOGNE, GERMANY • 15-16 AUGUST 2016 




Compression In Practice 1 


Reducing 

Network 

Traffic 
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Reducing Network Traffic 

HTTP 1.1 has compression built in 

Likely already available to you 
Only GZIP widely supported 

Google is pushing BROTLI 

Hardware support available 
Just turn it on! 
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Reducing Network Traffic 

A closer look at the data 

HTML, JSON, XML,... compress well 
Human readable -> low entropy 
Different from data in memory 
Conversion wastes CPU / memory 
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Reducing Network Traffic 

Data treatment options 

Omit whitespace and comments 
Separate static from dynamic data 
Transfer static data once (or never) 
i.e. replace Strings with IDs 
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Reducing Network Traffic 

Use binary data formats 

i.e. MsgPack, ProtoBuffers, Binary XML,... 
Ditch HTTP, TCP/UDP have less overhead 
If HTTP, consider WebSocket support 
Beware: Base64 re-adds ~25% 
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Reducing Network Traffic 

Faster compression options 

Free: LZ4, Density 
Commercial: LZO, Selkie, LZB16 
Much (!) faster than GZIP 
Lower to equal compression ratio 
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Reducing Network Traffic 

Stronger compression options 

Free: ZStd, BROTLI 
Commercial: Mermaid 
Faster decompression speed 
Slower to equal compression speed 
Equal to higher compression ratio 
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Reducing Network Traffic 

General hints 

Make compression configurable 

If on HTTP, turn HTTP compression off 

Encrypt after compression 

Beware compressor memory overhead 

Make use of streaming, when possible 
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Reducing Network Traffic 

Teh Future 

HTTP/2 will be a binary protocol 


Shared dictionaries 

SDCH or home made (i.e. using ZStd) 
BROTLI has a generic dictionary built in 
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Compressing small data : with Dictionary 



Perfpipe records 
~ 300 bytes 
(max : 500 bytes) 



Mercurial events 
~ 1.5 KB 
(max : 64 KB) 



ESJSON docs 
~ 6 KB 

(max : 16 KB) 


■ Direct 

■ Dictionary 


Source: http://zstd.net 
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Compression In 


Shrinking 

Download Size 


Practice 2 



COMPRESSED 

GAME 

V J 
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Shrinking Download Size 

Game asset downloads of all kinds 

HTTP is usually a must (CDN) 

HTTP overhead insignificant 
HTTP compression not optimal 

Data is rarely changed 

-> Use strongest compression available 


GDC 




GAME DEVELOPERS CONFERENCE EUROPE COLOGNE, GERMANY • 15-16 AUGUST 2016 


Shrinking Download Size 

Compression Options 

Free: LZMA, XZ, LZHAM 
Commercial: LZNA, Kraken, BitKnit 
Slow to very slow compression 
Very high ratios 
Slow to fast decompression 
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Shrinking Download Size 

General Hints 

Consider keeping files compressed locally 
HTTP request delays and limits 

Few big files > many small files 
Use parallel downloads, if possible 

Don't forget about decompression time 
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Compression In Practice 


Creating 

Small App 

Packages 




TappyChicken 

97.07MB 


rrVi Flappy Bird 





4.06MB 
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Creating Small App Packages 

Why is it different? 

Platform owners enforce package format 
.apk, .ipa, .appx, ... 

Actually just .zip files 

Built in compression far from optimal 

-> Compress before packaging 



Creating Small App Packages 

Textures 

Best compression: JPEG (or similar) 

Most pitfalls: PNG 

Don't use Photoshop output for final images! 

Use compressed texture formats if possible 

Don't forget to apply regular compression 

Consider custom image format 

1 1 1 1 
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Creating Small App Packages 

Textures - The Future 

RDO - Rate-distortion optimization 

https://qithub.com/BinomialLLC/crunch 

Transcoding between compressed formats 
New compressed GPU formats 
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Creating Small App Packages 


Geometry & Animation 

Highly format dependent 
Strip unneeded data 

Tangents, Binormals, Extra Uvs,... 

Lossy animation compression 
Compress using a generic algorithm 
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Creating Small App Packages 


Sound and Music 

Use lossy compression 

MP3, Ogg/Vorbis, BINKA, ... 

Depends on audio platform 

Check back with provider 

Consider mono for music 
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Creating Small App Packages 


Config, Settings, Loca,... 

Use generic algorithm 

BROTLI is aimed at text 

Consider binary formats 

Convert at packaging time 



LIVING 


LA VIDA LOCA 




Creating Small App Packages 


Further complications 

Certain files have fixed formats 

App icons, splash screens, ... 

Exe is encrypted / signed 

Consider interpreted code 

Lobby platform owners? 


gSJS; 
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Conclusions 

Take care of your data from day 1 
There is more than Deflate / Zlib \ 


Smaller data makes people happy! 
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Resources 


Yann Collet 

Blog: http://fastcompression.bloqspot.conn/ 

LZ4: http://cyan4973.qithub.io/lz4/ 

ZStd: http : //www . zstd . n et / 

Oodle 

Official: http://www.radqametools.com/oodle.htm 
Charles Bloom: http://cbloomrants.bloqspot.com/ 
Fabian Giesen: https://fqiesen.wordpress.com/ 
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Resources 

BROTLI 

Standard: https://www.ietf.org/rfc/rfc7932.txt 
Source: https://qithub.conn/qooqle/brotli 
Misc 

Rich Geldreich (LZHAM): http://richq42.bloqspot.com/ 
Binomial: http://www.binomial.info/ 

LZO: http://www.oberhumer.com/ 

7z / LZMA / XZ: http://www.7-zip.org/ 

Density: https://qithub.com/centaurean/density 
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Contact Information 

dietmar.hauser@sproinq.com 
https://qithub.com/Rattenhirn 
@ Ratten hirn 

https://www.sproinq.com 

https://fb.me/sproinq 

@SproingGames 



